How can I delay notification while I wait to see if it resolves?

#1

We have a system using Solarwinds that does health checks on our app every five minutes.

When we get a “Down” event, I’d like to trigger a PagerDuty alert. We have all that set up and working. But we get occasional false alerts, which resolve the next time (five minutes later).

I’d like to set up the notifcations to trigger the alert, but not start the call chain for 10 minutes. If the “Up” email arrives, we’d auto-resolve the event, and no one has to wake up.

How can we get that to happen? I can’t figure out how to insert a step in the escalation policy to “do nothing for ten minutes”

2 Likes
#2

Hi Eric,

We don’t currently have a feature for delayed alerting, but this does come up from time to time. I will submit a feature request to our product team for this.

In the meantime, the best workaround we can suggest would be to create a “dummy” user to be assigned to the incident immediately when it is triggered. Since you want to wait 10 minutes to see if an “Up” email arrives, you can set it your escalation policy to escalate to the second layer of the escalation policy after 10 minutes.

This dummy user would be on-call on the first level of the escalation policy by themselves, and would be immediately assigned to triggered incidents until the incident escalates. The reason you need to create a dummy user is because at this time, all open incidents must be assigned to a user.

Please let me know if you have questions about the above set up.

1 Like
#3

Note that this came up recently in /forum/t/delay-escalation-for-set-time-period/1800

You’ll want a separate Service and Escalation Policy for these alerts so everything else doesn’t get caught in the “wait-and-see” case. It should be named obviously enough that people don’t reassign to it trying to get a real person.